LHA Tarawa
|
Posted - 2010.08.17 20:21:00 -
[1]
I work for a small company you may have heard of... IBM. Well, we were just bought by them. Before IBM we were a little company you never heard of called Initiate Systems.
Long-story short, we take billions of records (could be medical records such as doctor visits, billing, perscription, insurance; could be government records such as tax records, dirver's licenses, birth records, tickets, police incident reports (crime reports); heck we even do bank transactions, phone calls, Point of Sale transations....)
Anyway, we take these billions of records and then run a probabilistic scoring and matching, highly configurable algorithm to link records that are really the same person, place, thing... We then build relationships between the objects...
Oh, sounds easy right? Wrong.
A new record comes in, we have score that record against the billions of records already in the system to see where it goes. Then we have to check to see if that new record has changed the relationship between the object that it belogs to and all other objects in teh system.
So, what's my point. Yeah, this stuff is hard.
We can tell the customer to get more servers to process transactions, but that just shifts the bottleneck to the database. So, we have them get faster disk arraies, more memory for cache, and better design their databse schema... Which puts the bottleneck on our ETL brokers that pre-process transations on the way in and out....
Then they get so fast that they are bringing in so many new records that the hash buskets are getting too large more frequently, so we make them redo their frequency counts and rebucket more often which... (okay, getting too technical here)
In short, there is always a bottleneck. When we fix one bottleneck, the next bottleneck it revealed.
AND, this is hard stuff. As mentioned in the dev blog.... trying to log performance against millions of transactions creates gigs of log data that can't be sifted and create their own performance issues.
There are some REALLY basic things you know to do, and assuming you've done those your performace is okay. Getting above "just okay" is where it gets really, really hard.
In the end, if performance really isn't what teh customer was hoping because they have too complicated of a matching and scoring algorithm, eventually we have to tell the customer, sorry, but you are simply going to have to limit the candidate set and the complexity of teh scoring algorithm....
EVE's equivilant would be to simply limit how many characters/ships/drones... whatever's are allowed to be on the same grid. Now, work that into a coheasive game design!
Guess we could go "turned based".... Instead of your guns firing every x seconds, they fire every x melee rounds.... Ugh, token ring instead of ethernet style communication... SUCK.
|